DOCUMENT RESUME 



ED 460 309 



CG 029 136 



AUTHOR 

TITLE 

INSTITUTION 

SPONS AGENCY 
PUB DATE 
NOTE 
CONTRACT 
AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Austin, Bryn 

A Supplement to Understanding Evaluation: The Way to Better 
Prevention Programs . A College Case Study. 

Higher Education Center for Alcohol and Other Drug 
Prevention, Newton, MA. 

Department of Education, Washington, DC. 

1997-00-00 

32p.; For companion document, see ED 361 604. 

SS95013001 

Higher Education Center for Alcohol and Other Drug 
Prevention, Education Development Center, Inc., 55 Chapel 
St., Newton, MA 02458-1060. Tel: 800-676-1730 (Toll Free) ; 
Fax: 617-928-1537; Web site: http://www.edc.org/hec/; 
e-mail : HigherEdCtrOedc . org , 

Reports - Research (143) -- Tests/Questionnaires (160) 

MF01/PC02 Plus Postage. 

Administrators; Athletes; Behavior Standards; Case Studies; 
College Faculty; College Students; Cooperation; Curriculum; 
*Drinking; *Drug Use; Evaluation; Fraternities; Higher 
Education; Mass Media Effects; Outcomes of Education; Peer 
Groups; * Prevent ion; Rape; Social Behavior; Sororities; 
Violence 



ABSTRACT 

This case study was developed as a supplement to the U. S. 
Department of Education publication Understanding Evaluation: The Way to 
Better Prevention Programs. Its purpose is to help readers get a feel for 
what is involved in setting up an evaluation of a college alcohol and other 
drug (AOD) prevention program. Although Understanding Evaluation specifically 
addresses prevention work in secondary schools and presents a case example of 
the fictional Wood County School District to illustrate the evaluation 
process, the principles of evaluation discussed are applicable to schools at 
any level, including the college or university campus. The fictional Woodson 
College case study is a composite of the evaluation experiences of a number 
of colleges and universities woven together to help readers tailor the 
information in Understanding Evaluation to meet their own campus needs. In 
the case example, administrators at Woodson College instruct the director of 
prevention services to develop an evaluation plan for the Woodson's three AOD 
prevention programs. One program is a curriculum infusion project, the second 
is a social norms media campaign, and the third is a peer education project. 
Discusses and evaluates each of the three projects. Provides a partial list 
of publications and fact sheets/prevention updates available from the Higher 
Education Center for Alcohol and Other Drug Prevention. (Author/MKA) 



Reproductions supplied by EDRS are the best that cam be made 
from the original document 



o^ 

o 

cn 

o 

VO 



Q 



W 



A College 



Case Study 



A SUPPLEMENT TO 

Understanding Evaluation: 
The Way to Better 
Prevention Programs 



Bryn Austin 

Harvard School of Public Health 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 




(T 

rO 

o 

o 

C ■<> 

ERIC 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



A publication of the Higher Education Center 
for Alcohol and Other Drug Prevention 
Funded by the U.S. Department of Education 

BEST COPY AVAILABLE 



2 




A College Case Study 



For further information contact: 

The Higher Education Center for Alcohol and Other Drug Prevention 

Website: http://www.edc.org/hec/ 

Phone: (800) 676-1730 
Fax: (617) 928-1537 
E-mail: HigherEdCtr@edc.org 



This publication was produced with funding from the U.S. Department of Education under contract number 
SS95013001 with Education Development Center, Inc. Views expressed are those of the contractor. No official 
support or endorsement by the U.S. Department of Education is intended or should be inferred. 



1997 

The Higher Education Center for Alcohol and Other Drug Prevention 
Education Development Center, Inc. 

55 Chapel Street 

Newton, Massachusetts 02158-1060 
Cover design and layout by Kay Baker 




3 



A College Case Study 



Contents 

Preface v 

Introduction vii 

Woodson College Broadens its 

Alcohol and Other Drug Prevention Prograiraning 1 

A Comprehensive Prevention Plan 2 

Objectives and Outcomes 3 

Initial Reports from the Directors 3 

Curriculum Infusion Program 3 

Social Norms Media Campaign 5 

Peer Education Program for Intercollegiate Athletes 
and Fraternity and Sorority Members 6 

Intended Outcomes for Overall Prevention Strategy 7 

Choosing Measures 10 

Evaluation Measures Proposed by the Directors 10 

Curriculum Infusion Program 10 

Social Norms Media Campaign 10 

Peer Education Program for Intercollegiate Athletes 
and Fraternity and Sorority Members 1 1 

Assessing Impact 13 

Establishing That Implementation of the Programs Took Place 16 
Evaluating Student Outcomes 19 

Notes 23 

Center Publications 24 



A College Case Study 



Preface 



The Higher Education Center for Alcohol and Other Drug Prevention 
was established by the U.S. Department of Education in 1993 to assist 
institutions of higher education in developing and carrying out alcohol 
and other drug (AOD) prevention programs that will promote campus 
and community safety and help nurture students' academic and social 
development. 

To accomplish this mission, the Center seeks to increase the capacity 
of postsecondary schools to develop, implement, and evaluate programs 
and policies that are built around environmental management strate- 
gies. Environmental management means moving beyond general aware- 
ness and other education programs to identify and change those factors 
in the physical, social, legal, and economic environment that promote or 
abet alcohol and other drug problems. 

Clearly, stemming the use of alcohol and other drugs is not some- 
thing that college administrators alone can achieve. Top administrators, 
especially presidents, must exercise leadership, but their success will 
depend ultimately on their ability to build a strong coalition of both on- 
campus and community interests. The better AOD prevention programs are 
campuswide efforts that involve as many parts of the college as possible, 
including students, staff, and faculty. For this reason, the Center 
emphasizes team-focused training and technical assistance work. 

Building coalitions with local community leaders is also key. College 
campuses do not exist in isolation. AOD prevention planners need to col- 
laborate with local leaders to limit student access to alcohol, prevent 
intoxication, and support the efforts of local law enforcement. The 
Center therefore seeks to motivate and train academic leaders to work 
with local community representatives, while also joining with national 
organizations that urge local coalitions to increase their outreach to aca- 
demic institutions. 

Specific Center objectives include promoting (1) college presidential 
leadership on AOD issues; (2) formation of AOD task forces that include 
community representation; (3) reform of campus AOD policies and pro- 
grams; (4) a broad reexamination of campus conditions, including acade- 
mic standards and requirements, the campus infrastructure, and the 
academic calendar; (5) formation of campus-community coalitions that 
focus on environmental change strategies; and (6) the participation of 
individuals from the higher education community in state -level and 
other associations that focus on public policy. The Center also seeks to 
increase the capacity of colleges and universities to conduct ongoing 
process and outcome evaluations of AOD prevention activities, both on cam- 
pus and in the surrounding community. 
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This publication represents one piece in a comprehensive approach to 
AOD prevention at institutions of higher education. The concepts and 
approaches it describes should be viewed in the broader context of preven- 
tion theory and the approaches affirmed by the U.S. Department of 
Education and promoted by the Center in its training, technical assistance, 
publication, and evaluation activities. 

For information on Center services, please contact: 

The Higher Education Center for Alcohol and Other Drug Prevention 
Education Development Center, Inc. 

55 Chapel Street 
Newton, MA 02158-1060 
Tel.: (800)676-1730 
Fax: (617)928-1537 
Website: http://www.edc.org/hec/ 

E-mail: HigherEdCtr@edc.org 
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Introduction 



The following case study was developed as a supplement 
to the U.S. Department of Education publication 
Understanding Evaluation: The Way to Better Prevention 
Programs. Its purpose is to help readers get a feel for 
what is involved in setting up an evaluation of a college 
alcohol and other drug (AOD) prevention program. While 
Understanding Evaluation specifically addresses preven- 
tion work in secondary schools and presents a case 
example of the fictional Wood County School District to 
illustrate the evaluation process, the principles of evalua- 
tion discussed are applicable to schools at any level, 
including the college or university campus. 

The fictional Woodson College case study is a composite 
of the evaluation experiences of a number of colleges and 
universities woven together to help readers tailor the 
information in Understanding Evaluation to meet their 
own campus needs. In the case example, administrators 
at Woodson College instruct the director of prevention ser- 
vices to develop an evaluation plan for the Woodson’s three 
AOD prevention programs. One program is a curriculum 
infusion project, the second is a social norms media cam- 
paign, and the third is a peer education project. The case 
example walks readers through the steps Woodson College 
takes to design and implement a comprehensive evalua- 
tion of its AOD prevention programs. With this illustra- 
tion in hand, prevention specialists, administrators, and 
others concerned with preventing AOD use on college 
campuses should find Understanding Evaluation a prac- 
tical guide in planning evaluations for their own campuses. 
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Woodson College Broadens Its Alcohol and 
Other Drug Prevention Programming and 
Strengthens Its Evaluation Strategy 



t Woodson College, a midsize institution of 9,000 students 
with a mix of residential and commuter students, administra- 
tors were becoming increasingly aware of problems both on 
and off campus resulting from students' alcohol and other drug 
(AOD) use and associated violence. Recently a group of its students 
was arrested after a bar brawl in town, and a high-profile sexual 
assault charge against a top male athlete had led to a barrage of 
negative publicity about the school. Although these incidents were 
more extreme than usual for the campus, administrators and health 
program planners realized they would be missing the mark if they 
saw them as the isolated acts of a few problem students. 

Health educators in several areas of the college, including the 
student health clinic and the athletic department, had begun to 
develop plans to implement new AOD prevention programs, but 
administrators were beginning to think that more work was needed 
to coordinate efforts across the different parts of the campus and to 
evaluate the programs. The recent incidents, the administrators 
agreed, were indicators that it was time to reassess the AOD-related 
climate on campus and develop a more comprehensive prevention 
strategy. And because they were committing the additional 
resources and staff time to launch broad-based prevention programs, 
they also wanted an evaluation plan to determine how well the 
programs were implemented and how effective they were in changing 
AOD use and associated violence on campus. 

Once the administrators had reached consensus on the need for 
evaluation, they issued a memo to the head of prevention services, 
Ms. Anderson, outlining their expectations for an ongoing assess- 
ment of the scope of program implementation and tracking of AOD 
use on campus. They wanted to see a formal evaluation plan within 
three months and a report on results at the end of each academic 
year. Beyond these instructions, they left it up to Ms. Anderson to 
come up with the best strategy for assessing the outcomes of the 
programs. 





A College Case Study 



A Comprehensive Prevention Plan 




^^^WJeveral months before the administration called for more a con- 
^^ft^certed evaluation effort, Ms. Anderson had attended a regional 
BB^w^conference on campus AOD prevention programs and met with a 
number of other educators to share ideas and experiences relating to 
prevention programming on their campuses. After the conference, 
Ms. Anderson and her staff conducted some follow-up research to 
find out more about programs tried on campuses outside of their 
region and around the country and settled on three programs that 
appeared to be especially promising and well suited to the needs of 
Woodson College. 

The first component of Woodson’s three-pronged prevention 
strategy was a curriculum infusion program, where AOD prevention 
content was integrated into the academic curriculum in a range of 
courses offered on campus. The second, a social norms approach, 
was designed to reduce binge drinking rates by correcting the wide- 
spread misperception that the majority of college students tj^ically 
binge drink. The third was a peer education program, where leaders 
from student groups on campus with the heaviest alcohol consump- 
tion patterns — intercollegiate athletes and Greek society members — 
were trained to conduct prevention workshops with teammates or 
fraternity and sorority members. ^ 

Ms. Anderson and her staff believed these three programs well 
suited for Woodson for a number of reasons. Several professors from 
different departments had expressed interest in taking a more active 
role in AOD prevention because of concerns about acquaintance rape 
and other interpersonal violence on campus. These professors were 
willing to advocate for curriculum infusion in their departments. A 
campus-based media campaign was likely to have a broad reach 
among students at Woodson because the majority of students read 
the college paper and either lived on campus or spent considerable 
time in campus buildings. And a peer education program targeting 
the Greek societies and athletic teams looked promising because the 
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20 fraternity and sorority houses on campus sponsored a large num- 
ber of the social activities at the college, and several of the athletic 
teams were well known on campus and popular among the students. 

Acting on the administrators’ instructions, Ms. Anderson set up 
a meeting with the directors of the three programs to brief them on 
the college administration directive for a coordinated evaluation 
strategy. The administration, she explained, wanted baseline infor- 
mation collected on alcohol and other drugs on campus, assessment 
of each of the three programs, and documentation of any changes 
resulting from the programs, including reduction in violence. She 
appointed the three directors to the new evaluation team and asked 
each of them to write up a short summary of their objectives and 
intended short-term outcomes specific to their program. 

Objectives and Outcomes 



he three program directors and Ms. Anderson met a week later 
to discuss initial ideas on their program objectives and outcomes. 
The brief reports presented by the program directors follow. 

Initial Reports from the Directors 

Through a curriculum infusion strategy, prevention messages were 
being incorporated into regular course work in a variety of academic 
departments on campus.2 For instance, sociology and media courses 
covered how the social norms of drinking were shaped by advertising 
and images of alcohol in other popular media. Criminal justice class- 
es addressed ways that alcohol contributed to domestic and other 
interpersonal violence. Likewise, other departments incorporated 
prevention themes as they related to the standard curriculum. 
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Curriculum Infusion 
Program 



3 



10 




A College Case Study 



Ideally, teachers either tested students on the prevention material 
or had them write papers on a related topic. Professors in six 
departments plus the director of the required freshman writing 
seminar volunteered to collaborate with the curriculum infusion 
program. The six collaborating departments were Sociology, 
Psychology, Communications, Government and Political Science, 
Women’s Studies, and African American Studies. 



Objectives 

♦ In the first year, collaborate with six departments to have at 
least one course in each department infused with AOD pre- 
vention curriculum. One of the papers assigned in the required 
freshman writing seminar should be on an AOD prevention 
topic. 

♦ In the first year, enroll 2,000 students in the freshman writing 
seminar and 1,000 students in infused courses in the six collabo- 
rating departments. 

♦ In the second year, have each department offer a second infused 
course and add three more collaborating departments. 

♦ In the second year, expose all 2,000 students in the freshman 
writing seminar and 2,000 students in infused courses in collab- 
orating departments. 

Short-term Outcomes 

♦ Increase participation of professors as leaders in AOD prevention. 

♦ Increase student opportunities for education on AOD risks and pre- 
vention. 
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The Woodson College social norms media campaign was 
designed to challenge the widespread misperception among students 
that the vast majority of their classmates engaged in binge drink- 
ing.3 In a survey conducted the previous year, health educators had 
found that while 43 percent of the students binge drank, students 
believed that the binge drinking rate on campus was 70 percent. 

The educators had read other research suggesting that perceptions 
of binge drinking influence actual binge drinking — if students 
believed everyone else was doing it, they would be more likely to do 
it themselves. In response, Woodson College created a social norms 
media campaign to correct the misperception. 

The campaign relied on print media — including advertising and 
editorial space in the college paper, press releases, flyers, and 
posters — to reach the majority of students. The campaign also 
included a series of minicontests, where hired student actors work- 
ing in pairs approached small groups of students in the cafeteria 
and other gathering spaces to hold ad hoc AOD knowledge contests. 
Students who correctly stated the true drinking prevalence on cam- 
pus each won a prize of $5. In addition to the contest, the student 
actors incorporated minisessions on AOD prevention into their routines. 



Objectives 

♦ Place two editorials per semester on AOD prevention issues in 
the campus paper. 

♦ Have two articles per semester on AOD prevention issues in 
the campus paper. 

♦ Purchase 10 pages of advertising space per semester for the 
campaign in the campus paper. 

♦ Place posters about drinking norms on every floor of every dorm 
and in student gathering places such as cafeterias and lounges. 

♦ Train four student actors to conduct educational contests 
throughout the school year. 
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♦ Conduct 50 educational contests per semester in campus gather- 
ing places. 

Short-term Outcomes 

♦ Change student perception of drinking prevalence on campus. 

♦ Enhance campus social environment supporting light drinking or 
abstinence. 



Peer Education 
Program for 
^ Intercollegiate 
Athletes and 
Fraternity and 
Sorority Members 




In collaboration with the Woodson College athletic department 
and the campus Greek society governing council, health educators 
developed a peer alcohol and other drug prevention education pro- 
gram. While athletes and fraternity and sorority members were not 
the only students at the college drinking, they did, on average, drink 
more and more often compared with other students. Educators felt 
they would be best reached with a targeted prevention program. 
Varsity athletes and Greek society members were recruited to take 
on the role of health opinion leaders and were trained to conduct 
AOD prevention workshops with their teammates and in their fra- 
ternity and sorority houses. They also were responsible for organiz- 
ing regular alcohol-free social events on campus, such as movie 
nights, concerts, and coffeehouses. 

Objectives 

♦ Train two peer educators from each of the 10 sororities and 10 
fraternities on campus. 

♦ Because so many men participate in football, train two peer edu- 
cators from the varsity squad and two from the junior varsity squad. 

♦ Train an additional five women and five men from the various 
other sports teams to be peer educators. 

♦ Conduct at least two AOD prevention workshops per season with 

13 




A College Case Study 



each sports team and two per year in each fraternity and sorority 

house. 

♦ Sponsor 10 alcohol-free social events per year, each attended by 
at least 50 students. 

Short-term Outcomes 

♦ Increase leadership role of athletic director, coaches, and Greek 
society council in AOD prevention. 

♦ Increase AOD prevention participation of peer leaders in athletics 
and Greek societies. 

♦ Increase opportunities for athletes and fraternity and sorority 
members to learn about AOD risks and prevention. 

♦ Increase opportunities to participate in alcohol-free social events. 

♦ Enhance environmental support for light drinking or abstinence 
among student athletes and Greek society members. 



intended Outcomes for Overall Prevention Strategy 

Satisfied with the objectives and intended short-term outcomes 
defined by the three directors, Ms. Anderson led the evaluation 
team members in a brainstorming session to come up with addition- 
al intended outcomes for the overall prevention strategy. The group 
agreed on the following short-term, intermediate, and long-term out- 
comes for the three-pronged program as a whole: 
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Short-tenn Outcomes 

♦ Increase in students’ knowledge of risks of AOD use 

♦ Change in students’ attitudes toward AOD use 

Intermediate Outcomes 

♦ Reduction in AOD use 

♦ Reduction in violence and other harm associated with AOD use 

♦ Decreased costs associated with vandalism and property damage 

♦ Improved academic performance 

♦ Improved physical, mental, social, and emotional health status of all 
students 

Long-term Outcomes 

♦ Improved academic and social development of all students and 
enhanced campus and community safety 

♦ Improved preparation of students to become good citizens and 
develop a civil society 

The following page illustrates the various components of 
Woodson’s overall AOD prevention strategy. At the conclusion of the 
session, Ms. Anderson asked the directors to consider the following 
two questions. The first question, she explained, was the basis of a 
process evaluation and the second was an essential starting place for 
an outcome evaluation. At their next meeting they would begin to 
hash out their evaluation measures. 

♦ What information do you need to be able to assess whether your 
program is being implemented as it was intended? 

♦ What information do you need to determine whether the pro- 
gram is making a difference? 
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LOGIC MODEL FOR WOODSON COLLEGE CAMPOS AOD PREVENTION PROGRAMS 



Strategies 






Objectives 



(a) Curriculum Inimion 
Program 



J 



(b) Social Norms 
Media Campaign 






Curriculum Infusion Program 

collaborate with six departments and 
freshman writing seminar program in 
first year 

reach 2,000 students in freshman writing 
seminars and 1,000 in other courses 
infused with prevention content 
have each collaborating department 
offer a second infused course in second 
year 

add three more collaborating depart- 
ments in second year 
reach 3.000 students in courses infused 
with prevention content in second year — 
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Social Norms Media Campaign 

place two editorials per semester on AOD 
prevention issues in campus paper 
have two articles per semester on AOD pre* 
vention issues in campus paper 
purchase 10 pages of advertising space per 
semester for the campaign in the campus 
paper 

place posters on drinking norms on every 
floor of eveiy dorm and in student gath- 
ering places 

train four student actors to conduct 
minieducational contests 
conduct 50 educational contests per 
semester in campus gathering places 
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(c) Peer Education 
Program 
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(c) Peer Education Program 

* train two peer educators from each of the 
10 sororities and 10 fraternities on campus 

* train two peer educators from the varsity 
football squad and two from the junior 
varsity squad 

* train an additional five women and five 
men from the various other sports 
teams 

* conduct at least two AOD prevention 
workshops per season with each sports 
team and two per year in each fraternity 
and sorority house 

► sponsor 10 alcohol-free social events per 
year.e 
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Short-term Outcomes 

(a) Increased participation of 
professors as leaders in AOD 
prevention 

(b) Increased student opportu- 
nities for education on AOD 
risks and prevention 

(c) Change in student percep- 
tion of drinking prevalence 
on campus 

(d) Enhanced campus social 
environment supporting light 
drinking or abstinence 

(e) Increased leadership role of 
athletic director, coaches, 
and Greek society council in 
AOD prevention 

(0 Increased participation by 
peer leaders in athletics and 
Greek societies in AOD pre- 
vention 

(g) Increased opportunities for 
athletes and fraternity and 
sorority members to learn 
about AOD use risks and pre- 
vention 

(h) Increased opportunities to 
participate in alcohol-free 
social events 

(i) Enhanced environmental 
support for light drinking or 
abstinence among student 
athletes and Greek society 
members 

(j) Increase in students’ knowl- 
edge of risks of AOD use 

(k) Change in students' attitude 
toward AOD use 



Intermediate Outcomes 

(a) Reduced AOD use 

(b) Reduced violence and other 
harm associated with AOD 
use 

(c) Decreased costs associated 
with vandalism and property 
damage 

(d) Improved academic perfor- 
mance 

(e) Improved physical, mental, 
social, and emotional 
health status of all students 



Long-Term Outcomes 

(a) Improved academic and 
social development of all 
students and enhanced 
campus and community 
safety 

(b) Improved preparation of 
students to become good 
citizens and develop a civil 
society 



Implementation 



Outcome 
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Choosing Measures 




ach member of the evaluation team came to the meeting with a 
list of both potential process evaluation and outcome evaluation 
measures specific to the needs of his or her program. 



Evaluation Measures Proposed by the Directors 



Curriculum 
Infusion Program 



The most important curriculum infusion evaluation measures were as 
follows: 

Process 

♦ Number of courses with prevention content 

♦ Number of class sessions with prevention content 

♦ Number of faculty participating in infusion 

♦ Number of departments participating in infusion 

♦ Number of students enrolled in courses with infusion 

♦ Assessments of infusion program by participating faculty 

♦ Assessments of infused curriculum by exposed students 

Outcome 

♦ Changes in knowledge, attitudes, and behavior of students 
exposed to infused curriculum 



Social Norms Media 
Campaign 



As with any media campaign, the first step in an evaluation is to 
determine what proportion of the target audience is actually being 
reached. An evaluation of the campaign would, therefore, require 
the following information: 
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Process 

♦ Number of editorials, articles, and advertisements placed in 
campus paper 

♦ Number of posters displayed in dorms and gathering places on 
campus 

♦ Number of students exposed to the media message promoting 
moderation 

♦ Frequency of exposure to the media campaign 

♦ Students’ perception of the credibility of the media message 

♦ Number of student actors trained 

♦ Number of contests conducted 

Outcomes 

♦ Students’ level of recall of media message 

♦ Students’ perception of binge drinking rates on campus 



For an evaluation of the peer education program, the following 
information would be needed: 

Process 




♦ Number of peer educators trained 

♦ Number of teams, fraternities, and sororities represented by the 
educators 

♦ Number of workshops conducted by the peer educators 

♦ Number of students attending the workshops 

♦ Number of alcohol-free social events sponsored 

♦ Number of students attending the social events 



Peer Education 
Program 




18 



A College Case Study 



♦ Assessment of program by peer educators 

♦ Assessment of program by students attending workshops and 
alcohol-free events 

Outcome 

♦ Change in knowledge, attitudes, and behavior of students attend- 
ing the workshops 

After hearing the presentations from each director, Ms. Anderson 
agreed that the process measures recommended by the program 
directors were on target but that the outcome measures needed more 
consideration: She pointed out the need for an evaluation strategy 
that would not only assess factors specific to each program but also 
evaluate outcomes relevant to all three programs. Because many 
students on campus were likely to be exposed to two or even three of 
the programs over the same period, it would be difficult in most cases 
to link changes in alcohol-related knowledge, attitudes, or behavior 
to any one program. In addition, she explained, it would be impera- 
tive that they collect baseline information on student AOD use and 
associated violence before program implementation to be able to show 
before and after comparisons. 

Ms. Anderson suggested that at the next meeting they discuss 
what information was needed to determine the overall impact of the 
alcohol and other drug prevention efforts on campus and how they 
could collect these data in a way that would minimize cost and staff 
burden. 
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t the next meeting, Ms. Anderson and the program directors 
debated the pros and cons of several possible indicators of 
impact, but in the end decided that they would keep the evalua- 
as simple and clearly focused as possible, choosing the two out- 
comes they considered the most important. They agreed that a solid 
evaluation of the impact of prevention efforts required them to track 




tion 



Assessing Impact 



♦ patterns of alcohol and other drug use by students 

♦ alcohol- and other drug-related incidents, including acquain- 
tance rape and other interpersonal violence, reported to cam- 
pus authorities 

The incident data were already being logged by the campus 
police, so culling this information from security records involved min- 
imal work for the prevention staff. To ensure confidentiality, preven- 
tion staff summary reports would not include any identifying 
information about victims or other students involved in incidents 
reported to campus police. 

Tracking alcohol and other drug consumption would be more 
complicated. In preparation for the meeting, Ms. Anderson 
researched how drinking patterns had been assessed on other cam- 
puses and found that the task involved careful planning in terms of 
which survey instrument to select, when to administer it, and how to 
include a representative sampling of the student body. 

In reviewing survey instruments used by other colleges, Ms. 
Anderson found the Core Survey, developed by the Core Institute 
based at the Center for Alcohol and Other Drug Studies at Southern 
Illinois University, to be a top choice.-* This instrument was designed 
for large campuswide surveys and had been used at hundreds of col- 
leges around the country. A short self-report instrument (it takes 
only 20 minutes to complete), it had been scientifically validated for 
use with college populations. The Core Institute had developed a 
detailed user’s manual and a 15-minute videotape explaining how to 
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use the instrument. Also, the institute staff offered telephone and 
online technical assistance and would conduct data analysis of survey 
results for any school administering the Core Survey instrument. 

Ms. Anderson realized that, in order to be able to claim that 
changes seen at Woodson College were due to the prevention pro- 
grams, the team would need to estimate with some degree of accuracy 
what drinking patterns would have been like in the absence of the 
programs. One way to do this was to use a pretest and posttest eval- 
uation design — that is, survey the students both before the programs 
and again a year later and then assess whether there had been any 
changes over the period the programs were up and running. The 
pretest/posttest design would lend some credibility to claims of the 
programs’ effectiveness if changes in drinking patterns had occurred. 
It would not preclude the possibility, however, that the changes would 
have occurred anyway regardless of the prevention program. In other 
words, how would the team demonstrate the programs’ effectiveness 
if Woodson College were to find that drinking rates on campus 
decreased during the time of the prevention programs, but that at the 
same time rates had fallen on campuses all across the country where 
prevention programs were not in place? Members of the evaluation 
team decided that they needed to identify, at the start date, other col- 
leges with student demographics and drinking patterns similar to 
Woodson’s but without extensive prevention programs, so that any 
changes in drinking and other drug use patterns could be compared 
in subsequent years. An additional bonus of using the Core Survey 
would be that where many other schools had used the measure, 
Woodson’s data could easily be compared with data from other univer- 
sities. 

A third important consideration in tracking drinking on campus 
was that the survey be administered to a representative sample of 
students. With 9,000 students enrolled at Woodson, the evaluation 
team felt it would be too costly and burdensome to survey everyone. 
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Plus, they knew that, to provide them with statistically sound and 
representative information, it would be less costly and cumbersome 
to use random sampling techniques to survey a subset of students. 

On the advice of a statistician, they decided to send copies of the 
Core Survey to a random sample of 1,000 students.® Using complete 
listings of student names and addresses, they would assemble a rep- 
resentative sample by choosing a starting place on the college enroll- 
ment list at random, then selecting every ninth student until 1,000 
students had been identified. For a college the size of Woodson, a 
random sample of 1,000 students would be large enough to have sta- 
tistical power and to accrue sufficient students of color to allow some 
comparison across race/ethnicity categories. 

The surveys would be anonymous, but response postcards with 
unique identifiers would be attached so that Ms. Anderson and the 
evaluation staff could target nonresponders with follow-up mailings 
and calls. To further increase the survey response rate, the evalua- 
tion team decided to give cash awards and prizes to a small number 
of respondents selected by lottery as an incentive to complete the 
surveys. Their goal was to achieve at least an 80 percent response 
rate. 

With their program objectives and short-term, intermediate, and 
long-term outcomes defined and their process and outcome measures 
specified, Ms. Anderson and the evaluation team were confident they 
were ready to report back to the college administration with their 
evaluation plan. The next task would be to carry out the baseline 
data collection using the Core Survey so that implementation of 
Woodson’s comprehensive AOD prevention programming could begin. 
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Establishing That Implementation of 
the Programs Took Place 




or each of the three programs, Ms. Anderson and her team set- 
tled on a series of process measures they believed would be 
important to a thorough evaluation. Because process measures 
usually need to be very specific to the program under evaluation, 
they created their own to collect the information they needed to 
determine whether the three programs were implemented as intend- 
ed. The following charts and forms are examples of those used at 
Woodson College for the process evaluation of each prevention program. 



Individual Course Record for Curriculum Infusion Program 
Fall Semester 



Course title and number American History, 1945 to the Present (H201) 



Professor 

Department 

Number of students enrolled 
Number of class sessions infused 
with prevention content 
Estimated number of class hours 
infused with prevention content 
Type and number of course 
assignments on prevention content 



Camilla Chang 

History 

47 

2 

1.5 hours 

1 3-page paper, 4 questions on midterm 




Curriculum Infusion Program Record 
Fall Semester 



Total number freshman writing seminars with prevention content 17 (out of 20) 



Total number of other courses with prevention content 6 

Total number of departments participating 7 

Total number of faculty participating 10 

Total number of students enrolled in infused courses 1 ,834 
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Social Norms Campaign Record 
Fall Semester 



Number of editorials in campus paper 2 

Number of articles in campus paper 1 

Number of advertisements in campus paper 7 

Number of posters displayed in dorms and campus gathering places 68 



Percentage of Students Who Recall Seeing Social 
Norms Campaign in Campus Media 

80 
70 
60 
50 

Percent 40 
30 
20 
10 
0 




I 

Fall1999 Spring 

2000 
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Participants' Peer Education Process Evaluation Feedback Form 

Please indicate whether you agree or disagree with each of the 
following statements by placing an X in the appropriate box. 




Agree 

strongly 


Agree 


Neither 

agree 

nor 

disagree 


Disagree 


Disagree 

strongly 


The information was 
presented clearly. 












The information was 
provided at the 
appropriate level of 
detail. 












The peer educator 
held my interest. 












The written and 
visual materials were 
easy to understand. 












Any other comments? 








Peer Education Program Record 
Fall Semester 



Total number of peer educators trained 25 

Total number of athletic teams represented by peer educators 5 
Total number of fraternities and sororities represented 13 

Total number of workshops conducted by peer educators 17 

Total number of students attending workshops 372 
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Evaluating Student Outcomes 



For the outcome evaluation, Ms. Anderson's team had decided to 
use the well-established Core Survey to measure patterns of alcohol 
and other drug use by students over time and to compare use rates 
with other colleges. They also planned to track alcohol- and other 
drug-related incidents reported to the campus authorities to see if 
the number of incidents decreased once the programs were imple- 
mented. See page 20 for examples of questions included on the 
Core Survey that Woodson students were to answer. 

Once their outcome evaluation was underway, the team were 
able to collect data to compare alcohol and other drug use patterns 
of Woodson students with patterns at other colleges before the pre- 
vention programs were launched and also to assess some program 
impact. On pages 21-22 are examples of charts and graphs the 
Woodson College evaluation team created to represent the results 
of their outcome evaluation. 

As illustrated in the fictional Woodson College case study, eval- 
uating campus AOD prevention programs is a complex process and 
requires careful planning, implementation, and follow-up. For 
more guidance on how to evaluate college AOD prevention efforts, 
readers should contact the Higher Education Center for Alcohol and 
Other Drug Prevention (see back cover). 
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?0 

ERIC 



17. Within the last ^ 

about how often have ^ 

you used... ^ 

^madr one for each line} ^ 

a. Tobacco (smoke, chew, snuff) . . 

b. Alcohol (beer, wirte, liquor) OOMOOj^QjOO 

c. Marijuana (pot, hash, hash oil) 000-0 O'^iBWjQO 

d. Coc^ne {cradK, rock, freebase) 00,®00-0<300 
e? Amphetamines (diet pills, speed) O0^JOO-Qi^QOCI) 

f. Sedatives (downers, ludes) 0'^^^0Q©j©!0 O 

g. Hallucinogens (LSD, POP) .... OC^^pOpOpO 

h. Opiates (heroin, smack, horse) G'^^,OO^0f)O 

I. Inhalants (glue, solvents, gas) . . QiQ0O O 

j. Designer drugs (ecstasy, MDM A)O.C4^iO 0:Q£X0 O 

k. Steroids O|©%OO;@0;OO 

l. Other illegal drugs OS^ OO0l©|OO 



18. During the past 30 days 
on how many days 
did you have: 

(mark one for each line) 

a. Tobacco (smoke, chew, snuff) . . 

b. Alcohol (beer, wine, liquor) 

c. Marijuana (pot, hash, hash oil). 




:0mo0m 



d. Cocaine (crack, rock, freebase) 

e. Amphetamines (diet pills, speed) 

f. Sedatives (dovmers, ludes) 



.#,@po:a@o 

.m.0oo'0mo 

Mdod&mo 



modommo 

mQOOBMO 

mdoGdmo 



g. HallucIrKigens (LSD, PCP) 

h. Opiates (heroin, smack, horse) 

i. Inhaiants (glue, solvents, gas) . . 



j. Designer drugs (ecstasy. MDMA) 

k. Steroids 

l. Other illegal doigs 



MPpOGdo 



MaoooMo 



19.HOW often do you ^ 

think the avera^ student ^ 

on your campus uses. .. \ 

(mark one for each line} % 

a. Tobacco (smoke, chew, snuff) . , O0JSOO^©|OO 

b. Alcohol (beer, wine, liquor) .... 000000000 

c. Marijuana (pot, hash, hash oil) Q G0Q Q Q 0Q O 
d Cocine (crac^, rock, freebase) O00OOp»OOO 

e. Amphetamines (diet pills, speed) (D0MOO0OOO 

f. Sedatives (downers, ludes) 0.0 OO OttoO O 

g. Hallucinogens (LSD, POP) .... 0ppp0p<^00 

h. Opiates (heroin, smad<. horse) ^ r\ 

i. Inhalants (glue, solvents, oas) . . 

j. Designer drugs (ecstasy, MDMA 

k. Steroids 

). Other iliegal drugs 



odmocm&oo 

)Oppooppoo 

oo:ooop,#Do 




20. Where have you 
used... 

^marff all that apply} 

a. Tobacco (smoke, chew, snuff) . 

b. Alcohol (tear, wine, liquor) . . . 

c. Marijuana (pot, hash, hash oil) 

d. Cocaine (cradc, rock, freebase) 

e. Amphetamines (diet pills, sp8ed)O00OOS30OO 

f. Sedatives (downers, ludes) QQ0Q 0!0 0!0 O 

gThallucinogens (LSD* PCP) .... OOS^OOpOlOO 
h. Opiates (heroin, smack, horse) O00O 0000 O 
L Inhalante (glue, solvents, gas) . , Q00O Q00Q Q 

j. Designer drugs (ecstasy, MD^)O0 ” 

k. Steroids 0|C 

l. Other illegal drugs 



OiCpOOfe^QO 

opppoppoo 



22. Have any of your family had alcohol or other 
drug problems: (mark all that apply} 

O Mother O Brothers/sisters O Spouse 

O Father O Mother’s parents O Children 

O Stepmother O Father’s parents O None 

O Stepfather O AuntsAincles 



21. Please indicate how often 
you have experienced 
the following due to 
your drinking or drug use 
during the last year... 

(mark one for eac^ line} 

a. Had a hangover 

b. Performed poorly on a test 

or important project 

c. Been in trouble ^th police, 
residence hall, or other 

college authorities 

d. Damaged property, pulled 

fire alarm, etc 

e: Got into an argument or fight 

f. Got nauseate or vomited 

g. Driven a car white under 

the influence 

h. Missed a class 

i. Been criuctzed by someone 

I know * 

j. Thought I might have a drinking 

or other drug problem 

k. Had a memory loss 

l. Done something I later regretted . . 

m. Been arrested for OWI/DUl 

n. Have been taken advantage 

of sexually 

o. Have taken advantage 

another sexually 

p. Tried unsuccessfully to stop using 

q. Seriously thought about suicide . . 

r. Seriously tried to commit suicide . . 

s. Been hurt or Injured 




. . oo&j>po 

. . OOP'^"' 

. . odi^oo 

. . ooSdoo 
. . ooiaoo 
..oopioo 

..ooMoo 
. . ooggoo 

. . oopaoo 

..oofegoo 
. . ooedoo 
.. oa#poo 
. . oopgoo 

. . ooi^dbo 

si 

..OOMOO 
. . OOPOOO 

..oopdoo 

..OOPOOO 

. . oogpoo 



23. If you volunteer any of your time on or off campus 
to help odiers, please Indicate the approximate 
number of hours per month and prtnclpal activity: 
O Don’t volunteer, or O 10-15 hours 
less than 1 hour O 16 or more hours 

O 1-4 hours Prirrdpal volunteer activity is; 

O 5-9 hours 
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Number of Drinks per Week at Woodson College 


and 66 Colleges Using Core Survey 






Percent of Students at 


Percent of Students at 


Number drinks per week 


Woodson College 


66 Colleges Using Core Survey 




(n=1,012) 


(n=36,627) 


None 


32.5 


36.8 


One 


13.7 


14.8 


2 to 5 


25.2 


23.3 


6 to 9 


13.0 


7.6 


10to 15 


10.6 


10.4 


16 to 20 


1.8 


2.9 


21 or more 


3.2 


4.3 
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Percent of Students Using the Most Frequently Used Drugs "in the Last Year" 


Drug 


Woodson College 


66 Colleges Using Core Survey 




(n=1,012) 


(n=36,100) 


Alcohol 


87.2 


85.9 


Tobacco 


35.6 


39.6 


Marijuana 


32.8 


24.4 


Hallucinogens 


4.3 


5.5 


Amphetamines 


4.7 


5.4 


Cocaine 


2.5 


3.4 



Average Number of Drinks per Week, listed by Grade Point Average 




A B C D/F 
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Notes 



‘ More information about these three prevention strategies are presented in 
the following publications available from the Higher Education Center for 
Alcohol and Other Drug Prevention: Making the Link: Academics and 
Prevention (in press) by Barbara Ryan and William DeJong; A Social Norms 
Approach to Preventing Binge Drinking at Colleges and Universities (1996) 
by Michael P. Haines; and Alcohol and Other Drug Prevention: A Bulletin for 
Fraternity and Sorority Advisers by Tammy Enos and Tanutda Pittayathikhun. 

2 This section on curriculum infusion draws on the work of the Network for 
Dissemination of Curriculum Infusion, presented in the monograph 
Characteristics of Successful Curriculum Infusion Programs: Analysis 
Conducted by the Network for Dissemination of Curriculum Infusion^ 
Northeastern Illinois University, 1996. 

3 This discussion of a social norms strategy is based on the Higher Education 
Center for Alcohol and Other Drug Prevention publication A Norms 
Approach to Preventing Binge Drinking at Colleges and Universities (1996) 
by Michael P. Haines. 

^ More information on the Core Survey and copies of the questionnaire are 
available through the Center for Alcohol and Other Drug Studies, Student 
Health Programs, Southern Illinois University— Carbondale, Carbondale, IL 
62901-6802; phone (618)453-4420. 

5 The survey methodology used in this case study is based on that designed by 
Wechsler and colleagues for the Harvard College Alcohol Study and described 
in the survey methodology guide Preventing Alcohol-Related Problems on 
Campus: Methods for Assessing Student Use of Alcohol and Other Drugs by 
William DeJong and Henry Wechsler and available from the Higher Education 
Center for Alcohol and Other Drug Prevention. 
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Prevenf/on'/s to ass/5fl/flsf/twr;ons'Ofe 
higher educMiort in a§f^^lpping . ■ v-|.^ 
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;ir ; ^>i MmiMe . Cm.Me lD 



Jhe Centfrioffers an integrated array of services to help people at colleges 
"and tiniverSities adopt effective AOD prevention strategies: ;;’• “ 



Resources, referrals, and consultations 



Support" for the Network of Colleges and Universities 
Committed to the Elimination of Drug and Alcohol Abuse 

Assessment, evaluation, -and analysis activities 



Keep"up^ to dat^ Catalyst. Learn about important developments in 

AQp prevention in higher education. To receive free copies, ask to be put 
otTsdurmaiim .v T- 4;- .■' ■ ■■ ' 



Get, injouch 



Additional information can be obtained by contacting: ^T *' 

The Higher Education Center for Alcohol and Other Drug Prevention 

Education Deyeloprnent Center, Inc. : 

55 Chapel. Street ■' ■ " ' . ' ' ' ■■> ' 

Newton, MA 02158-1060 ^ ‘ \ - 

Website; http://www.edc.org/hec/ - ^ ■ 

Phone: 80Q-676-1730; Fax: 617-928-1537 ' : 

E-mail; ' ■ HigherEdCtr@edc.org- .i'-;";;'--'- .... y/\. ;■ ' -TI.T; 
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Understanding Evaluation 
The Way to Better 
Prevention Programs 






The Higher Education Center 

for Alcohol and Other Drug Prevention 
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